hi!
I have the following problem. I need to read data from a TXT file our
company receives.
I would use StreamReader, and process it line by line using ReadLine,
however, the following problem occurs.
The file contains characters with ASCII codes above 128. But the file is
still text (nothing like UTF7/8 or the like). It also might contain + signs.
As a result:
UTF8 encoding doesn't read characters above 128
UTF7 encoding reads everything ok, except eats the + signs, and some
characters after them
ASCII encoding reads the + sign ok, however, characters above 128 are
disappear.
Because the file arrives in this form, I do not have any control on how it
looks like. The best idea so far was to create an own ReadLine method, that
reads the file byte after byte, and converts using UTF7, while taking
special care to feed the + character (ASCII code 46) to an ASCII encoder.
This way I could build a string from a line, that contains exactly what's in
the file.
But would there be a nicer way, or just this do-it-yourself-manually?
thanx
-Lenard 5 15006
Lenard Gunda <fr****@fbi.hu> wrote: I have the following problem. I need to read data from a TXT file our company receives. I would use StreamReader, and process it line by line using ReadLine, however, the following problem occurs.
The file contains characters with ASCII codes above 128.
No it doesn't, because there are no such things. ASCII is a 7-bit
encoding.
But the file is still text (nothing like UTF7/8 or the like).
UTF-7 and UTF-8 are text encodings - a file containing text in UTF-8
encoding is still a text file.
It also might contain + signs. As a result:
UTF8 encoding doesn't read characters above 128 UTF7 encoding reads everything ok, except eats the + signs, and some characters after them ASCII encoding reads the + sign ok, however, characters above 128 are disappear.
Because the file arrives in this form, I do not have any control on how it looks like. The best idea so far was to create an own ReadLine method, that reads the file byte after byte, and converts using UTF7, while taking special care to feed the + character (ASCII code 46) to an ASCII encoder. This way I could build a string from a line, that contains exactly what's in the file.
But would there be a nicer way, or just this do-it-yourself-manually?
It sounds like you really need to know what encoding your file is
*really* in. Have you tried Encoding.Default?
See http://www.pobox.com/~skeet/csharp/unicode.html for more
information.
--
Jon Skeet - <sk***@pobox.com> http://www.pobox.com/~skeet
If replying to the group, please do not mail me too
> UTF-7 and UTF-8 are text encodings - a file containing text in UTF-8 encoding is still a text file.
Yup I know that. I wanted to mean plain text file, one that can be read
without conversion. It sounds like you really need to know what encoding your file is *really* in. Have you tried Encoding.Default?
Well, it contains ASCII characters, extended-ascii characters that are (in
this case) Finnish language characters. There's probably a code-page number
that would describe it. But ... Encoding.Default solved my problem, so it
would seem. Thanks very much!
-Lenard
Lenard Gunda <fr****@fbi.hu> wrote: UTF-7 and UTF-8 are text encodings - a file containing text in UTF-8 encoding is still a text file. Yup I know that. I wanted to mean plain text file, one that can be read without conversion.
There's *always* conversion involved. The file is binary data, and you
want text data. There's a conversion involved, even if it's ASCII. It sounds like you really need to know what encoding your file is *really* in. Have you tried Encoding.Default?
Well, it contains ASCII characters, extended-ascii characters that are (in this case) Finnish language characters.
"Extended-ascii" isn't a well-defined character set (there are many
character sets which are extensions of ASCII) and anything above 127 is
*not* ASCII.
There's probably a code-page number that would describe it. But ... Encoding.Default solved my problem, so it would seem. Thanks very much!
To find out the code page of Encoding.Default, just look at
Encoding.Default.CodePage.
I'm glad it's working for you though.
--
Jon Skeet - <sk***@pobox.com> http://www.pobox.com/~skeet
If replying to the group, please do not mail me too
"Jon Skeet [C# MVP]" <sk***@pobox.com> wrote in message
news:MP************************@msnews.microsoft.c om... Lenard Gunda <fr****@fbi.hu> wrote: UTF-7 and UTF-8 are text encodings - a file containing text in UTF-8 encoding is still a text file.
Yup I know that. I wanted to mean plain text file, one that can be read without conversion.
There's *always* conversion involved. The file is binary data, and you want text data. There's a conversion involved, even if it's ASCII.
It sounds like you really need to know what encoding your file is *really* in. Have you tried Encoding.Default?
Well, it contains ASCII characters, extended-ascii characters that are
(in this case) Finnish language characters.
"Extended-ascii" isn't a well-defined character set (there are many character sets which are extensions of ASCII) and anything above 127 is *not* ASCII.
There's probably a code-page number that would describe it. But ... Encoding.Default solved my problem, so
it would seem. Thanks very much!
To find out the code page of Encoding.Default, just look at Encoding.Default.CodePage.
And note that it's working because your default encoding is the one with the
Finnish characters. If you need it to work on machines where this is not
the case, take Jon's advice: look at Encoding.Default.CodePage, and add
codes that explicitly uses that encoding to read the file.
Hi, And note that it's working because your default encoding is the one with
the Finnish characters. If you need it to work on machines where this is not the case, take Jon's advice: look at Encoding.Default.CodePage, and add codes that explicitly uses that encoding to read the file.
Yup, I finally managed to understand how these Encoders work, and found it
how to create one for a particular code page. Could be useful in the future,
but because this product is supposed to run on our server, it will have the
correct settings. But good advice, anyway.
Thanks for the help.
-Lenard This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: Richard Lewis |
last post by:
Hi there,
I'm having a problem with unicode files and ftplib (using Python 2.3.5).
I've got this code:
xml_source = codecs.open("foo.xml", 'w+b', "utf8")
#xml_source = file("foo.xml",...
|
by: Catalin Constantin |
last post by:
i have the following code:
c=chr(169)+" some text"
how can i utf8 encode the variable above ?
something like in php utf8_encode($var);?!
chr(169) is the © (c) sign !
10x for your...
|
by: Spamtrap |
last post by:
I only work in Perl occasionaly, and have been searching for a
solution for a conversion, and everything I found seems much too
complex.
All I need to do is take a simple text file and copy...
|
by: Wes Groleau |
last post by:
I have a file containing thousands of Spanish words, encoded AFAIK)
in UTF-8. I also have a perl script in UTF-8, which says (hope
pasting works):
#!/usr/bin/perl -w -CSD
#
# NOTE: The extra...
|
by: hunterb |
last post by:
I have a file which has no BOM and contains mostly single byte chars. There
are numerous double byte chars (Japanese) which appear throughout. I need to
take the resulting Unicode and store it in a...
|
by: Pohihihi |
last post by:
Hello NG,
I am making a small tool which reads files on harddisk and saves many
information about files in a db. Now, while reading information from the
file I want to figure out what type,...
|
by: Thomas Podlesak |
last post by:
I need a check, if a file is utf8 encoded. I only found the php-functions
'iconv' and 'recode'. But it seems it´s not possible to determine the
encoding with them. Isn´t there any similar...
|
by: weheh |
last post by:
I'm developing a cgi-bin application that must be unicode sensitive. I'm
striving for a UTF8 implementation. I'm running python 2.3 on a development
machine (windows xp) and a server (windows xp...
|
by: firepol |
last post by:
Hello there, I am dealing with files encoded in UTF8 and I can't find a way to convert them into ANSI.
I've already searched in google for this since a while, and I'm not achieving the result I...
|
by: emmanuelkatto |
last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud.
Please let me know.
Thanks!
Emmanuel
|
by: Sonnysonu |
last post by:
This is the data of csv file
1 2 3
1 2 3
1 2 3
1 2 3
2 3
2 3
3
the lengths should be different i have to store the data by column-wise with in the specific length.
suppose the i have to...
|
by: Hystou |
last post by:
There are some requirements for setting up RAID:
1. The motherboard and BIOS support RAID configuration.
2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing,...
| |